One way to store and process the three-dimensional sound field of spherical microphone arrays is the Higher-Order Ambisonics (HOA) representation. Ambisonics uses orthonormal spherical functions for describing the sound field in the area around and at the point of origin, or the reference point in space, also known as the sweet spot. The accuracy of such description is determined by the Ambisonics order N, where a finite number of Ambisonics coefficients are describing the sound field. The maximum Ambisonics order of a spherical array is limited by the number of microphone capsules, which number must be equal to or greater than the number O=(N+1)2 of Ambisonics coefficients.
An advantage of such Ambisonics representation is that the reproduction of the sound field can be adapted individually to nearly any given loudspeaker position arrangement.
Invention
While facilitating a flexible and universal representation of spatial audio largely independent from loudspeaker setups, the combination with video playback on differently-sized screens may become distracting because the spatial sound playback is not adapted accordingly.
Stereo and surround sound are based on discrete loudspeaker channels, and there exist very specific rules about where to place loudspeakers in relation to a video display. For example in theatrical environments, the centre speaker is positioned at the centre of the screen and the left and right loudspeakers are positioned at the left and right sides of the screen. Thereby the loudspeaker setup inherently scales with the screen: for a small screen the speakers are closer to each other and for a huge screen they are farther apart. This has the advantage that sound mixing can be done in a very coherent manner: sound objects that are related to visible objects on the screen can be reliably positioned between the left, centre and right channels. Hence, the experience of listeners matches the creative intent of the sound artist from the mixing stage.
But such advantage is at the same time a disadvantage of channel-based systems: very limited flexibility for changing loudspeaker settings. This disadvantage increases with increasing number of loudspeaker channels. E.g. 7.1 and 22.2 formats require precise installations of the individual loudspeakers and it is extremely difficult to adapt the audio content to sub-optimal loudspeaker positions.
Another disadvantage of channel-based formats is that the precedence effect limits the capabilities of panning sound objects between left, centre and right channels, in particular for large listening setups like in a theatrical environment. For off-centre listening positions a panned audio object may ‘fall’ into the loudspeaker nearest to the listener. Therefore, many movies have been mixed with important screen-related sounds, especially dialog, being mapped exclusively to the centre channel, whereby a very stable positioning of those sounds on the screen is obtained, but at the cost of a sub-optimal spaciousness of the overall sound scene.
A similar compromise is typically chosen for the back surround channels: because the precise location of the loudspeakers playing those channels is hardly known in production, and because the density of those channels is rather low, usually only ambient sound and uncorrelated items are mixed to the surround channels. Thereby the probability of significant reproducing errors in surround channels can be reduced, but at the cost of not being able to faithfully place discrete sound objects anywhere but on the screen (or even in the centre channel as discussed above).
As mentioned above, the combination of spatial audio with video playback on differently-sized screens may become distracting because the spatial sound playback is not adapted accordingly. The direction of sound objects can diverge from the direction of visible objects on a screen, depending on whether or not the actual screen size matches that used in the production. For instance, if the mixing has been carried out in an environment with a small screen, sound objects which are coupled to screen objects (e.g. voices of actors) will be positioned within a relatively narrow cone as seen from the position of the mixer. If this content is mastered to a sound-field-based representation and played back in a theatrical environment with a much larger screen, there is a significant mismatch between the wide field of view to the screen and the narrow cone of screen-related sound objects. A large mismatch between the position of the visible image of an object and the location of the corresponding sound distracts the viewers and thereby seriously impacts the perception of a movie.
More recently, parametric or object-oriented representations of audio scenes have been proposed which describe the audio scene by a composition of individual audio objects together with a set of parameters and characteristics. For instance, object-oriented scene description has been proposed largely for addressing wave-field synthesis systems, e.g. in Sandra Brix, Thomas Sporer, Jan Plogsties, “CARROUSO—An European Approach to 3D-Audio”, Proc. of 110th AES Convention, Paper 5314, 12-15 May 2001, Amsterdam, The Netherlands, and in Ulrich Horbach, Etienne Corteel, Renato S. Pellegrini and Edo Hulsebos, “Real-Time Rendering of Dynamic Scenes Using Wave Field Synthesis”, Proc. of IEEE Intl. Conf. on Multimedia and Expo (ICME), pp. 517-520, August 2002, Lausanne, Switzerland.
EP 1518443 B1 describes two different approaches for addressing the problem of adapting the audio playback to the visible screen size. The first approach determines the playback position individually for each sound object in dependence on its direction and distance to the reference point as well as parameters like aperture angles and positions of both camera and projection equipment. In practice, such tight coupling between visibility of objects and related sound mixing is not typical—in contrast, some deviation of sound mix from related visible objects may in fact be tolerated for artistic reasons. Furthermore, it is important to distinguish between direct sound and ambient sound. Last but not least, the incorporation of physical camera and projection parameters is rather complex, and such parameters are not always available. The second approach (cf. claim 16) describes a pre-computation of sound objects according to the above procedure, but assuming a screen with a fixed reference size. The scheme requires a linear scaling of all position parameters (in Cartesian coordinates) for adapting the scene to a screen that is larger or smaller than the reference screen. This means, however, that adaptation to a double-size screen results also in a doubling of the virtual distance to sound objects. This is a mere ‘breathing’ of the acoustic scene, without any change in angular locations of sound objects with respect to the listener in the reference seat (i.e. sweet spot). It is not possible by this approach to produce faithful listening results for changes of the relative size (aperture angle) of the screen in angular coordinates.
Another example of an object-oriented sound scene description format is described in EP 1318502 B1. Here, the audio scene comprises, besides the different sound objects and their characteristics, information on the characteristics of the room to be reproduced as well as information on the horizontal and vertical opening angle of the reference screen. In the decoder, similar to the principle in EP 1518443 B1, the position and size of the actual available screen is determined and the playback of the sound objects is individually optimised to match with the reference screen.
E.g. in PCT/EP2011/068782, sound-field oriented audio formats like higher-order Ambisonics HOA have been proposed for universal spatial representation of sound scenes, and in terms of recording and playback, a sound-field oriented processing provides an excellent trade-off between universality and practicality because it can be scaled to virtually arbitrary spatial resolution, similar to that of object-oriented formats. On the other hand, a number of straight-forward recording and production techniques exist which allow deriving natural recordings of real sound fields, in contrast to the fully synthetic representation required for object-oriented formats. Obviously, because sound-field oriented audio content does not comprise any information on individual sound objects, the mechanisms introduced above for adapting object-oriented formats to different screen sizes cannot be applied.
As of today, only few publications are available that describe means to manipulate the relative positions of individual sound objects contained in a sound-field oriented audio scene. One family of algorithms described e.g. in Richard Schultz-Amling, Fabian Kuech, Oliver Thiergart, Markus Kallinger, “Acoustical Zooming Based on a Parametric Sound Field Representation”, 128th AES Convention, Paper 8120, 22-25 May 2010, London, UK, requires a decomposition of the sound field into a limited number of discrete sound objects. The location parameters of these sound objects can be manipulated. This approach has the disadvantage that audio scene decomposition is error-prone and that any error in determining the audio objects will likely lead to artefacts in sound rendering.
Many publications are related to optimisation of playback of HOA content to ‘flexible playback layouts’, e.g. the above-cited Brix article and Franz Zotter, Hannes Pomberger, Markus Noisternig, “Ambisonic Decoding With and Without Mode-Matching: A Case Study Using the Hemisphere”, Proc. of the 2nd International Symposium on Ambisonics and Spherical Acoustics, 6-7 May 2010, Paris, France. These techniques tackle the problem of using irregularly spaced loudspeakers, but none of them targets at changing the spatial composition of the audio scene.
A problem to be solved by the invention is adaptation of spatial audio content, which has been represented as coefficients of a sound-field decomposition, to differently-sized video screens, such that the sound playback location of on-screen objects is matched with the corresponding visible location. Specifically, a method for generating loudspeaker signals associated with a target screen size is disclosed. The method includes receiving a bit stream containing encoded higher order ambisonics signals, the encoded higher order ambisonics signals describing a sound field associated with a production screen size. The method further includes decoding the encoded higher order ambisonics signals to obtain a first set of decoded higher order ambisonics signals representing dominant components of the sound field and a second set of decoded higher order ambisonics signals representing ambient components of the sound field. The method also includes combining the first set of decoded higher order ambisonics signals and the second set of decoded higher order ambisonics signals to produce a combined set of decoded higher order ambisonics signals and generating the loudspeaker signals by rendering the combined set of decoded higher order ambisonics signals. The rendering adapts in response to the production screen size and the target screen size.
The invention allows systematic adaptation of the playback of spatial sound field-oriented audio to its linked visible objects. Thereby, a significant prerequisite for faithful reproduction of spatial audio for movies is fulfilled.
According to the invention, sound-field oriented audio scenes are adapted to differing video screen sizes by applying space warping processing as disclosed in EP 11305845.7, in combination with sound-field oriented audio formats, such as those disclosed in PCT/EP2011/068782 and EP 11192988.0. An advantageous processing is to encode and transmit the reference size (or the viewing angle from a reference listening position) of the screen used in the content production as metadata together with the content.
Alternatively, a fixed reference screen size is assumed in encoding and for decoding, and the decoder knows the actual size of the target screen. The decoder warps the sound field in such a manner that all sound objects in the direction of the screen are compressed or stretched according to the ratio of the size of the target screen and the size of the reference screen. This can be accomplished for example with a simple two-segment piecewise linear warping function as explained below. In contrast to the state-of-the-art described above, this stretching is basically limited to the angular positions of sound items, and it does not necessarily result in changes of the distance of sound objects to the listening area.
Several embodiments of the invention are described below, which allow taking control on what part of an audio scene shall be manipulated or not.
In principle, the inventive method is suited for playback of an original Higher-Order Ambisonics audio signal assigned to a video signal that is to be presented on a current screen but was generated for an original and different screen, said method including the steps:
decoding said Higher-Order Ambisonics audio signal so as to provide decoded audio signals;
receiving or establishing reproduction adaptation information derived from the difference between said original screen and said current screen in their widths and possibly their heights and possibly their curvatures;
adapting said decoded audio signals by warping them in the space domain, wherein said reproduction adaptation information controls said warping such that for a current-screen watcher and listener of said adapted decoded audio signals the perceived position of at least one audio object represented by said adapted decoded audio signals matches the perceived position of a related video object on said screen;
rendering and outputting for loudspeakers the adapted decoded audio signals.
In principle the inventive apparatus is suited for playback of an original Higher-Order Ambisonics audio signal assigned to a video signal that is to be presented on a current screen but was generated for an original and different screen, said apparatus including:
means being adapted for decoding said Higher-Order Ambisonics audio signal so as to provide decoded audio signals;
means being adapted for receiving or establishing reproduction adaptation information derived from the difference between said original screen and said current screen in their widths and possibly their heights and possibly their curvatures;
means being adapted for adapting said decoded audio signals by warping them in the space domain, wherein said reproduction adaptation information controls said warping such that for a current-screen watcher and listener of said adapted decoded audio signals the perceived position of at least one audio object represented by said adapted decoded audio signals matches the perceived position of a related video object on said screen;
means being adapted for rendering and outputting for loudspeakers the adapted decoded audio signals.